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UjSTRACJT It is proposed that subjective contours are an artifact of the perception of natural three¬ 


dimensional surfaces, A recent theory of surface interpolation implies that “subjective surfaces” are con¬ 
structed in die visual system by interpolation between three-dimensional values arising from interpretation of 
a variety of surface cues. We show that subjective surfaces can take any form, including singly and doubly 
curved surfaces, as well as the commonly discussed fronto-parallcl planes. In addition, it is necessary in the 
context of computational vision to make explicit the discontinuities, both in depth and in surface orientation, 
in the surfaces constaicted by interpolation. It is proposed that subjective contours form tire boundaries of the 
subjective surfaces due to these discontinuities. Several novel subjective surfaces and subjective contours are 
demonstrated. The role played by figure completion and enhanced brightness contrast in the determination of 
subjective surfaces is discussed. All considerations of surface perception apply equally to subjective surfaces. 
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1. Introduction 


Under most circumstances, a visual contour is perceived when there is a relatively abrupt change in the 

irradiance received from adjacent areas of die visual field, due, for example, to a difference in brightness, local 

surface orientation, or color [sec, for example, the analytic studies of intensity profiles in Horn 1977, Marr 

* 

1976a, Binford 1981]. There are conditions, however, in which a relatively abrupt brightness gradient may be 
perceived in areas of the visual field where the physical stimulation is in fact roughly homogeneous. Schumann 


[1904] is usually credited with discovering such illusory contours, or “subjective contours” as they were named 


by Osgood [1953]. ITie study of subjective contours was popularized by Kanisza [1955], who presented many 


novel and compelling illustrations. Over the past twenty-five years they have been the subject of what Rock 


and Anson [1979] have termed a “flurry of research”. 


In spite of the high level of attention that has been devoted to subjective contours, there is no comprehen¬ 
sive single theory of their formation nor of the perceptual purpose that they serve. Among the more popular 
contenders as a theory are (i) Gestalt-like accounts of figural completion [Kanisza 1955, 1974, 1976, Osgood 
1953], (ii) various lateral inhibition accounts of enhanced brightness contrast that pay particular regard to the 
apparent brightness of subjective contours [Brigner and Gallagher 1974, Frisby and Clatworthy 1975, Day 


and Jury 1978, 1980, Jory and Day 1979, Kennedy and Lee 1976], (iii) interpretations of subjective contours 
as stacked planes [Coren 1972, Coren and Theodor 1975], which in all the authors’ examples are parallel to 
die frontal plane, with occlusion signalled by interposition , and (iv) a search for a cognitive interpretation 
[Gregory 1972, Rock and Anson 1979]. Some authors show a commendable spirit of compromise and contend 
that there is no single theory, rather die perception of subjective contours is inevitably multi-variate [Halpcrn 


1981]. 


With the exception of Corcn’s stacked plane interpretation, subjective contours are usually considered 
a part of two-dimensional figure perception. They are interpreted as the two-dimensional bounding contour 
of a figure, the competing theories implicitly assuming diilerenl accounts ol figure-ground separation. I he 
relationship between the perception of subjective contours and the perception of the three-dimensional 
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natural world is either not considered or is left unclear. In this article we develop an alternative account oi 
subjective contours that derives from the developing computational theory of vision [see Mari 1982, Gritnson 
1981a Ullman 1979c, Brady 1982 for overviews] and that places subjective contours squarely as a part of 
the perception of natural three-dimensional surfaces. In particular, as we discuss in more detail below, we 
propose that subjective contours mark discontinuities in surfaces interpolated from cues in an image [similar to 
the proposal made by Marr, 1982]. We do this by placing the computation of subjective contours within the 
context of the computation of subjective surfaces by interpolation from scattered cues in images. 


The processes of perception evolved to construct three-dimensional interpretations of the physical world 
from two-dimensional images. Although people have developed impressive capabilities for interpreting car¬ 
toons, text, and other “unnatural” images, figure perception draws heavily upon the resources of natural 
perception. So far as early visual processing is concerned (where by early processing we include such per 
ceptual abilities as edge finding, stereo, shape from texture, shape from shading, and shape from motion), 
information in the image is used to compute surface shapes with little or no regard to semantic or other 
“higher-level” attributes. That is, the cues derived from the image are assigned three-dimensional attributes 
without regard to the semantic structure of the final perception. Our interpretation of photographs and line 
drawings such as those shown in figure 1 attest to the “misapplication” of the processes of natural perception 
to two-dimensional figures. We have to be reminded that the drawings in figure 1 are in fact two-dimensional. 


so 


are 


strongly biased is our perceptual system toward three-dimensional interpretations. Io be sure, there 
figures such as those shown in figure 2 that are not interpreted in terms of three-dimensional surfaces; but that 
in no way implies that there is an entirely separate apparatus for figure perception, or that we do not sie/.c 
upon three-dimensional interpretations wherever possible. Indeed, an alternative statement of the legend to 
figure 2 is that it illustrates degenerate cases of surface perception, in which the three -dimensional surfaces 
depicted are parallel to the image plane. This interpretation is at the heart of Coren s account of subjective, 
contours, but for us it is merely a simple case of a more general phenomenon. 
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Figure 1. Two-dimensional figures given three-dimensional interpretations, (a) Two intersecting 
planes, (b) A cylinder. 




To form a three-dimensional interpretation, it is necessary to deploy information not given explicitly in 
die image. For example, although a wide variety of radically differing surfaces could give rise to the line 
drawings illustrated in figure 1, the surface shown in figure la is perceived as planar, while that in figure lb is 
cylindrical. The discovery and utilization of prior assumptions that provide the additional constraint needed 
to achieve this perception is a key part of the computational theory of vision. Importantly, such constraints 
can be of wide generality. For example, Binford [1981] has cataloged a number of constraints of the form “if 
two lines meet in an image at a corner, then the corresponding space curves meet in space.” Other constraints 
include constraints on stereo vision [Marr and Poggio, 1979], constraints on the irierprctation of occluding 


contours [Marr, 1977], constraints on surface interpolation [Grimson 1981b], constraints on motion interpreta¬ 


tion [Ullman, 1979a, c], constraints on interpretation of surface contours [Stevens, 1911 lc|, and Kanade’s 


discussion of skew symmetry and parallelism. 


I 






Figure 2. Two-dimensional figures given two-dimensional interpretations. 


The computational theory of vision, pioneered by Marr [1976a, 1976b, 1978, 1982] is outlined in section 
3. It first asks of a module of the visual system what is its purpose, or equivalently, what information it makes 
explicit. To date, considerable effort has been devoted in the field of computational vision to understanding 
stereo [Marr and Poggio 1979, Crimson 1981c, Mayhcw and Frisby 1981, Moravcc 1980], motion [Ullman 
1979a, Longuet-Higgins and Prazdny, 1981, Horn and Schunck, 1981], shape from shading [Horn 1975,1977, 
1981], shape from texture [Kendcr 1980, Stevens 1981b] and shape from contour [Stevens 1981c, Witkin 1981]. 
All of these modules make-explicit information about the layout of visible surfaces. We propose that subjective 
contours arise from the processes of natural surface perception , and in particular that subjective contours mark 
surface discontinuities. 


Wc introduce a recent theory of surface interpolation [Crimson 1981b, Brady and Horn 1981] in section 
3, In our account of surface interpolation, tile fu st things to he discovered arc discontinuities that eventually 
form pari ol the boundary of a * isiblc mm face. Note that the bound a to dooa no haw to K unnplcte: it is 



the subsequent completion of the bounding contour that gives rise to subjective contours. An interpolation 
process is then applied to construct a surface that is consistent with values hypothesized for certain points in 
the image. In general, many surfaces are consistent with a given set of boundary values and so additional 
constraint is required. Two constraints are proposed by tire theory. First, Grimson [1981b] observes that “no 
information is information”: since rapid changes in surface topography usually, give rise to visible edges and 
visible edges provide boundary values to the interpolation process, the absence of boundary values means that 
the underlying surface topography does not change too rapidly between discontinuities. This idea is made 
precise in the theory using tire calculus of variations to find the surface that is consistent with the given data 
and minimizes an appropriate performance index. Second, there is a technical requirement on the application 
of the calculus of variations that turns out to have an agreeable perceptual interpretation: the surface is 
interpolatedusing a rotationally symmetric operator that minimizes the quadratic variation of the surface. 'Ore 
computation by which this is effected can be implemented by a local parallel network of relatively simple 
computing elements [Grimson 1981a]. 


Notice that the interpolation process may not be totally constrained. The computation of surface dis¬ 
continuities precedes surface interpolation, which in turn precedes the computation of depth. The interpola¬ 
tion process determines the tilt and slant at all points on visible surfaces. The result may be underdetcr- 
mined, as in the Mach illusion. The computation of depth follows the interpolation process. It too may 
be undcrconstraincd. The Necker reversal is a familiar example of this phenomenon. The possibility for 
underconstraint in both depth and surface orientation finesses the arguments of Kennedy [1976a, page 110] 
and Rock and Anson [1979, page 666], both of whom assume that depth is computed directly and without 
ambiguity. 


Given that many difference cues can give surface information and that this-information is interpolated 
to form an explicit three-dimensional surface interpretation, this suggests that it should be possible to 
demonstrate subjective surfaces, with associated subjective contours at the discontinuities, due to each of the 
possible sui face cues. We do this in section Some prior evidence for this can be found in Coren [1971]. 
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In section 3 we outline the computational theory of vision, paying particular regard to surface interpolation, 
hich underlies the computation of subjective surfaces. In section 4 we make an initial attempt to construct a 


w 


c; 


atalog of three-dimensional constraints that arc hypothesized from certain image fragments, namely isolated 
points, line endings, and more complex figures. Finally, in section 5 we consider the relationship between edge 
detection and subjective contours, since variations on enhanced brightness have been offered as evidence in 
support of an early processing theory of subjective contours. 


2. The three-diinensionality of subjective contours 


The previous section sketched our approach to figure perception. If a figure has a three-dimensional 
interpretation, our perceptual system is strongly biased toward constructing it. We contend that a figure 
has a three-dimensional interpretation if and only if there exists a subjective contour with a similar three- 
dimensional interpretation. Figure 1 shows a set of line drawings whose preferred interpretation is three- 
dimensional. In each case there is a perceived internal discontinuity in suifacc orientation, a visible edge. 
According to our theory of surface interpolation, such discontinuities have to be made explicit, albeit in the 
form of additional subjective contours. Certain line drawings are interpreted as curved surfaces even though 

they have no internal edges. 

We intend to demonstrate first the variety of shapes that a subjective surface, and therefore a subjective 
contour, can take. We will then show how a wide variety of surface cues can drive the construction of a 

subjective surface and its associated subjective contours. 

Four such line drawings provide our first examples of subjective surfaces. Figure 3 shows a subjective 
surface that is usually perceived as a portion of a cylinder. Notice that the subjective contours lie on principal 
lines of curvature. It is in fact very difficult to draw a figure that is interpreted as a portion of a cylindrical 
surface in which the lines drawn do not correspond to lines of curvature. Figure 4 is a subjective version of the 
sail illusion. Although the usual interest in this figure stems from the fact that it has multiple interpretations, 
die interest here is the fact that ii has at least One. Figure 5 shows a subjective donor, and extends an example 



Figure 3. A portion of a subjective cylindrical surface. Notice that the subjective contours lie on 
principal lines of curvature. 



Figure 5. A subjective donut. As in figure 3, the lines that trigger the three-dimensional 
interpretation appear to lie along principal lines of curvature. 


discussed by Kennedy [1976a, figure 14, page 120]. As in figure 3, the lines that trigger die three-dimensional 
interpretation appear to lie along principal lines of curvature. Together with figure 6, which shows a subjective 
soap bubble, it demonstrates subjective surfaces that arc doubly curved. Figures 3 to 6 show that subjective 
surfaces need be neither planar, parallel to the image plane, nor singly curved, but in fact can correspond to 


X 


any type of complex surface. 




Figure 4. A subjective version of the sail illusion. As with most subjective contours, the effect 
can be made more compelling by increasing the contrast and viewing from a distance to narrow 

the visual angle. 



Tjmjre 7. A subjective hinge formed from two subjective surfaces. Sufficient cues have been given 
o enable the perceptual system to construct the subjective discontinuity in suriace orientation. 






Figure 6. A subjective soap film. Together with figure 5 this demonstrates that subjective surfaces 
do not need to be planar, parallel to the image plane, or even singly curved. 






Figure 8. A subjective staircase that builds upon the hinge in figure 7 and the Mach illusion. 
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Figure 10. When the right image is viewed with the right eye and the left image with Hie left 
eye. a planar white triangular surface is seen floating above the background. The triangular surface 
is seen as tilled about the median passing throught the bottom right vertex. Reproduced from 

Blomfield [1973, figure 2, page 256]. 


The next two examples require internal lines and illustrate the dictum 


“no information is information” 


discussed above. Figure 7 shows a subjective hinge between two surfaces. The discontinuity in surface orienta¬ 
tion has been marked with enough local cues to produce an additional subjective boundary. Figure 8 builds 


upon figure 7 and the Mach illusion to produce a subjective staircase. 


While most examples of subjective surfaces in the literature of subjective contours are planes parallel to 
the frontal plane, there are some examples of subjective surfaces that arc not parallel to the image plane. For 
example, Kennedy [1976b, figure 11, page 45] shows a stripe on a subjective cylinder (see figure 9). Figure 
10 is reproduced from Blomfield [1973, figure 2, page 256] and shows a stereo pair that fuses to yield a tilted 
surface. Figure 11 is the subjective Necker cube due to Bradley and Petry [1977, figure 1, page 254], From the 
standpoint of the theory advanced here, figure 11 is interesting for two reasons. 


First, the subjective surfaces that form the vertical sides of the Ncckcr cube are tilted relative to the image 
plane. Second, the example illustrates the point made earlier that the depth and orientation of an interpolated 
subjective sod u oi.ui he unrieiamMi aiiicd, yet give rise to the same subjective conf airs. 







Figure 9. A subjective stripe lying on a cylindrical surface. Reproduced from Kennedy [1976b, 
figure 11, page 45], 


In die previous section we noted that according to the computational theory of vision, the purpose of 
several modules of the visual system is to make surface layout explicit. Specifically, stereo, motion, texture 
discontinuities, and cues for interposition (sec for example Binford [1981]) provide evidence for surface layout, 
and can do so even in figures. In the rest of this section, we show that each of these cues can be the driving 
force behind the construction of a subjective surface. 


Figure 12 shows line drawings that are interpreted as the superposition of one figure on top of another. 
Figure 12a illustrates what we have been calling the degenerate case, in which the subfigurcs arc parallel to 
the image plane. Figure 12b illustrates that this is not a necessary condition. Figure 13a is the subjective 
version of figure 12a, originally discovered by Parks [1980a]. Figure 13b extends Parks demonstration and is 
the subjective version of figure 12b. Demonstrations of stacked subjective surfaces parallel to the image plane 
are given by Coren [1972] and Ivy figure 14, which is reproduced from Bradley and Duniais [1975, figure 3]. 






Figure 11. The subjective Necker cube invented by Bradley and Pelry [1977, figure 1, page 254]. 


Figure 15 is due to Kanisza [1974, figure 33, page 109] and shows a subjective surface that is parallel to the 
image plane and is framed by a tilted surface. In the same vein, figure 15b shows a ring surrounding tire top of 
a planar triangle. More significantly, figure 15c is based on an example of Kanisza and shows a tilted subjective 

surface that is framed by one that is parallel to the image plane. 


Subjective surfaces that result from stereo fusion are particularly interesting from die standpoint of the 
theory being developed here. Stereo has been much studied [for example, Wheatstone 1983,1952; Helmholtz 
1925, Julesz 1971] and has received considerable attention in the computational vision literature [for example, 
Marr and Poggio 1979, Crimson 1981c. Moravcc 1980, Mayhcw and Frisby 1981, Baker and Binford 1981], 
Stereo computes the angular disparity of matched edge feature points of two images and (up to scaling) this 
gives the relative depth at matched feature points. To obtain a complete surface representation, we can inter¬ 
polate between these known depth points (in. fact, the approach to surface intcipol ition outlined above was 
developed ill die contest of die output of stereo). Following the line of lese.tich )ioiie.acd by Jules/ one 





Figure 12. A line drawing interpreted as overlapping subfigures, (a) The subfigures are parallel 
to the image plane, (b) Neither subfigure is parallel to the image plane, (c) One subfigure is 
interpreted as parallel to the image plane, the other is not. 


would expect to find pairs of images, neither of which give rise to monocular subjective contours but in which 
subjective surfaces and bounding contours are found when the images are fused stcreoscopically. This has 
been confirmed on several occasions in a variety of ways, perhaps the most striking of which has been the work 
of Clulick and Lawson [1976, see also Lawson and Gulick, 1967, Blomficld 1973, Gregory and Harris 1974, 

Julcsz and Frisby 1975]. Julesz [1971] and Julesz and Frisby [1975] present a number of examples of subjective 

* 

contours that result from stereo fusion using random dot stereograms and anaglyphs. 

Gregory [1972a, page 52] reports an experiment in which the subfigures of a subjective contour example 
due to Kanisza [1955] are presented separately to each eye using a technique developed by Witasck [1899]. 
Subjective contours are seen following stereo fusing of the images. Gregory [1972b] measured apparent dis¬ 


parity with a light probe, and Gregory and Harris [1974] showed that measured disparity of a subjective stereo 
surface, corresponding to increased stratification, increases if and only if the pcrceied clarity of the subjective 
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Figure 13. (a) A subjective surface version of figure 12a, invented by Parks [1980, figure 3, page 
240j. (b) A subjective version of figure 12b. 

surface does. As we mentioned earlier, Blomfield [1973] demonstrated a tilted subjective surface when there 
is no monocular evidence for tilt. Finally, Stevens [1981a] reports a patient with visual agnosia who could 
not see subjective contours when they were presented monocularly, but who could see them when they were 


presented stereoscopically. 


The familiar cue of shape from shading [Horn 1977, 1981] is difficult to demonstrate for subjective sur¬ 
faces since a key attribute of subjective surfaces is the relative paucity of information given explicitly in the 
image to enable them to be constructed by the perceptual system. It is necessary to utilize more indirect kinds 
of information, such as shadow boundaries to generate subjective surfaces. Coren [1972, figure 3, page 361] 
presents a set of subjectively embossed letters that stand out from the page in intense side lighting. Figure 
16 shows a shadow on the sail illusion. Most observers find that it makes die subjective sail presented in 
figure 4 more compelling. Figure 17 illustrates a familiar point in the context of subjective surfaces. Figure 
]7a is a subjective figure that can be interpreted either as a two-dimensional pie sector or a three-dimensional 






Figure 14. The six-pointed star configuration discussed by Bradley and Dumais [1975, figure 3, 
page 583]. 


cone. Figure 17b shows that the interpretation is disambiguated by apparent shading that makes the surface 
curvature explicit. In this case the apparent shadow line fortunately lies along a line of curvature on the 


cone. 


Few subjective surfaces that arc produced by motion are reported. Motion is computed at several succes¬ 
sive stages by the visual system (sec for example Marr and Ullman [1980], Ullman [1979a,c]). Smith and Over 
[1979] show that motion after-effects can be produced as readily by exposure to subjective contours as by con¬ 
tours defined by irradiance discontinuities. The computation in question seems to be at the level of Ullman’s 
correspondence of primal sketch tokens. Ramachandran, Rao, and Vidyasagar [1973] also dcmonstiatc ap 
parent motion with subjective contours at the same level of processing. Parks [1980b] presents an example 
of motion perception involving subjective contours that seems to be at die level of surface pciception. When 
black disks arc moved to and fro, subjects perceive a continuous piece of “string” occluded by a subjective 

(planar) surface. 
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Figure 15. (a) A subjective surface that is parallel to the image plane and is framed by a tilted 
plane. Reproduced from Kanisza [1976, figure 33, page 109]. (b) A circular plane surrounding a 
subjective surface parallel to the image plane, (c) A tilted subjective surface that is framed by a 
planar surface parallel to the image plane. (Compare Kanisza [1974, figure 32, page 109].) 


Texture gradients have also been widely investigated as a cue to surface layout, starting with Gibson 
Corea [1972, figure 7, page 364] presents an example of a subjective contour formed by a texture 
gradient. Kennedy [1976a, figures 6 and 7 pages 112] (see figure 18) gives two examples of subjective inter¬ 
sections of tilted planar surfaces (figure 18). The tilt of the surfaces depicted in Kennedy’s figures is not com¬ 
pletely constrained. The figure illustrates die familiar convex-concave reversal. The image of the discontinuity 
is constrained however. 


In the previous examples we have constructed subjective surfaces with subjective contours as their 
boundaries and internal edges from the Mach illusion, the convex-concave reversal shown in figure 18, and the 
Meeker cube. This suggests that it might be possible to devise subjective correlates of so-called “impossible 
figures”: pe.iK lly possible firm, s, local fragments of which make three-dimensu cal sense, but which du not 




Figure 16. A subjective version of the sail illusion, made more compelling by the addition of 
“shadow” information. 



Figure 17. (a) A subjective figure that can be interpreted either as a planar pie shape or in three 
dimensions as a cone, (b) Apparent shading disambiguates the choice by making the curvature of 
the cone’s surface explicit. 











Figure 18. A sujective intersection of two planar surfaces whose perceived tilt derives from 
the texture gradient. The intersection is subject to convex-concave ambiguity. Reproduced from 
Kennedy [1976a, figure 6, page 112]. 


make global three-dimensional sense [Penrose and Penrose 1958, Draper 1981, Huffman 1971]. Coren [1972, 
figure 4, page 362] is aware of impossible figures, and presents a figure that appears to be shadow writing, 
but which is not globally consistent with regard to the position of die light source. The subjective contour 
that results is reported to be unstable. Similarly, Kanisza [1974, figure 34, page 109] presents a figure that 


appears to consist of a planar triangle intersecting a planar subjective triangle that is parallel to the image 


plane. Although this makes sense locally, it does not globally. Figure 19 is constructed using the technique of 


Bradley and Retry [1977, figure 1] and shows the impossible notched cube discussed by Draper [1980]. 


In summary, it appears that interpreting subjective contours as the boundaries of more or less constrained 
subjective surfaces is capable of explaining the vast majority of published demonstrations and the novel 
examples, using a wide lunge ul ' isual cues, presented in ibis section. ' 





Figure 19. A subjective contour forming an “impossible figure”, the notched cube of Draper 
[1980]. 


3. Outline of a computational theory of early vision. 


Having demonstrated a clear relationship between subjective contours and their interpretation as three- 
dimensional contours, we now turn to the consideration of a framework in which to explain this connection. 
The computational theory of vision, pioneered by Marr [1976a, 1976b, 1978, 1982], considers the goal of the 
early and intermediate human visual system to be the construction of several representations, each of which 
makes different visual information explicit. The primal sketch makes explicit information about the changes 
in image irradiancc, obtained by locating the zero-crossings in the convolution of the image with a set of 
V 2 G filters, where V 2 is the Laplacian, and G is a Gaussian distribution. The 2\-D sketch makes explicit 
information about the shapes and reflective properties of surfaces and their distribution relative to the viewer. 
A number of different v isual modules compute information about the 2 .^-1) sketch from the primal sketch, for 
example, stereo, motion, texture, and surface contours. As well as local surface si ipe being made explicit in 
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the 2A-D sketch, contours of surface discontinuities, either in depth or in surface orientation, are also made 
explicit. We have proposed that subjective contours arise from the process of natural surface perception at the 
level of the 2^-D sketch, and in particular, that subjective contours mark these surface discontinuities (similar 

to the proposal of Marr [1982]). 

Because all of the visual modules mentioned above use some form of the primal sketch as input, they 
compute explicit surface information only at a subset of the points in an image, corresponding to a subset 
of the zero-crossings. This implies that some form of surface interpolation must take place, to account for 
the perception of complete surfaces. A recent theory of surface interpolation [Crimson, 1981b, Brady and 
Horn, 1981] has proposed that the visual system computes a complete surface representation by constructing a 
surface which is consistent with surface values hypothesized for certain points in the image, for example, depth 
values computed by the stereo system along the zero-crossing contours. 

Since in principle any one of an infinite class of surfaces could fit through the known scattered stereo 
data, some additional constraint is needed to determine the correct interpolated surlacc. Grimson identified 
the surface consistency constraint , informally known as no news is good news (or no information is information ), 
which states: The absence of zero-crossings constrains the possible surfaces . 


This can be seen informally by the following argument. Suppose one had a closed circulai zeio-crossing 
contour, along which the depth was constant, and that there were no further zero-crossings interior to the 
circle. One possible interpolating surface would be a flat disk, a second would be a hemi-sphere. A thild 
would be the highly convoluted surface formed by a radial sine function. Note, however, that in the latter case, 
since the surface orientation is undergoing a rapid periodic variation, under most imaging situations the image 
irradiances will also undergo a periodic variation. But if this is true, then there should be additional interior 


zero-crossings, corresponding to those variations. Since there arc not, this surface is not acceptable. Hence, the 
lack of zero-crossings in a region of the image implies that there is no radical change in the suifuce shape, in 
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More formally, the surface consistency constraint can be rephrased using the calculus of variations to 

• « 

state that the required surface is one that is consistent with the given data and minimizes an appropiiate func¬ 
tional. Simply requiring that the functional measure surface consistency is not sufficient to guarantee a unique 
“best” surface fitting the known data. The addition of some minor constraints on the form of the functional, 
that it be a semi-inner product over a semi-Hilbert space, is sufficient to guarantee a unique solution suriace, 
up to possibly an element of tire null space of the functional. If a final requirement of a rotationally symmetric 
operator for minimizing the functional is added, it can be shown that there is an infinite family of possible 
functionals. This family forms a vector space [Brady and Horn, 1981], spanned by the quadratic variation: 


// 


fix + Vly + fly dxdy 


and the square Laplacian: 


f J i^fxx ~t~ fyy) dxdy. 




Finally, by considering the size and structure of the null space [Grimson, 1981b], or by considering the 
constraints supplied by the calculus of variations [Brady and Horn, 1981], it can be shown that the optimal 
functional is the quadratic variation. In other words, the surface perception problem is solved by finding the 
surface that fits through the known data, and minimizes the functional of quadratic variation elsewhere. A 
straightforward parallel, local-support algorithm can be devised for computing this surface [Giimson, 1981a]. 

Thus, the surface consistency constraint can be combined with a number of other simple constiaints to 
obtain an algorithm for the interpolated surface which is most consistent with the primal sketch descriptions of 
the image [Grimson, 1981a, 1981b]. The main point is that the algorithm finds the “most conservative” surface 
which fits the known data, introducing inflections or discontinuities in die surface only when forced to by the 
data (c.f. the notion of “fair surfaces” in car design [Forrest, 1972]). If there is no explicit evidence for a sharp 
change in shape, such as an inflection in the surface, then none is interpolated. On the other hand, the role 
of subjective contours in this context now becomes clear. Subjective contours are placed in the interpolated 
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surface representation along contours at which the interpolated surface undergoes a sharp change in shape, 
thus interpolating contours of discontinuity between scattered cues in the image. 

The crucial point of the surface consistency constraint is that an extended and continuous surface is to 
be constructed from a small set of cues. 1 here is a potential continuum of choices of finite sets of cues that 
could be made explicit; but not all of them lead to the perception of the surface. The surface consistency 
constraint and the interpolative view of surface perception of which it is a part, says that we can safely leave 
vast amounts of information out and still get the surface. We need to make explicit at least portions of certain 
discontinuities: of tangent to die bounding contour, of surface slope, indeed anything that could lead to a zero 
crossing in the processing of an image of the surface by the visual system. Clearly, the entire discontinuity 
need not be made explicit, else the entire subjective contour would have to be made explicit at this level. 


The important factor is that if no portion of a discontinuity is made explicit, the visual system conservatively 
assumes “no news is good news” and diat nothing is changing. 

To be sure, this set of cues is necessary but not necessarily sufficient. For a given set of cues, a variety of 
surfaces can be interpolated. If there are many such, the subjective surface is at most faintly perceived. As die 
surface interpolation becomes more constrained, perhaps by adding more constraints, the subjective surface 
is seen more persistently and more clearly. The perception of surfaces from natural images derives fiom a 
number of different types of cues, and the interpolation process has to be capable accepting such different 


sorts. Critically the cues are interpreted in drree dimensions not two. 

As an e xample of this process, we illustrate in figure 20 a random dot stereogram, and the interpolated 
surface that minimizes the functional of quadratic variation. The sharp discontinuities in depth, occuning 
between (he different planes of the stereogram arc clearly evident. 

Finally, several different algorithms for finding subjective contours suggest themselves: (1) Interpolate 
in two dimensions to complete a line drawing, dien interpret the line drawing as the image of a three- 
dimensional surface. The latter step could proceed by first interpreting the lines as three-dimensional space 

v. 

curves (see Harrow and Tenenbautn [19811). (2) Interpret the local cues directly in -Free dimensions to proceed 
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Figure 20, A random dot stereogram of a series of layered planes, and the interpolated surface 
which minimizes the quadratic variation of the surface. Reproduced from Grimson [1981a]. 


lo the “contour” drawing, then interpolate the surface between the given contours. (3) Proceed from local cues 
to detailed models of familiar objects, and use this knowledge to guide the filling in of the subjective surface. 

Our arguments above that subjective contours fill a natural role as discontinuities of three-dimensional 
surface information would tend to suggest that interpolation of line drawings in two dimensions is not com¬ 
putationally sound. Further, the existence of subjective contours in stereo perception, which arc not evident 
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in the monocular images yet are vividly present in binocular fusion (see for example figure 20 and [Gulick 
and Lawson, 1976, Lawson and Gulick, 1967]), argues against completing contours in the image before in¬ 
terpreting them in the surface representation. On the other hand, the ability to construct complete surface 
representations, including explicit surface discontinuities, of unknown objects or objects seen for the first time 
argues against using local cues to choose detailed models of familiar objects and then using this infoimation 
to guide the interpolation process. Thus, in the context of our computational arguments, the most likely algo¬ 
rithm is one in which local cues are directly interpreted in three dimensions, providing a “contour” drawing of 
surface discontinuities, within which the surface is interpolated. Note that while it may be possible to design 
an algorithm for constructing subjective contours that operates on two-dimensional image constructs, such an 
algorithm must have as its underlying basis, a three-dimensional theory. 


4. Some image cues and their interpretation 


In section 1 we argued that if a figure has a three-dimensional interpretation, our perceptual system is 

» 

strongly biased toward constructing it. In particular, the processes that evolved for natural three-dimensional 
early vision are “misapplied” to two-dimensional figures. By suggesting that the perception of natural surfaces 
is at the heart of understanding the purpose of subjective surfaces and contours, such as those illustrated in 
section 2, wc implicate all of the ideas discussed in section 3, including edge detection, stereo, structure from 
motion, shape from texture, and so on. Section 3 outlined a computational approach to early and intermediate 
vision and described a theory of surface perception.. By our account, the creation of subjective surfaces and 
subjective contours proceeds in three stages: (1) a set of cues are isolated in the primal sketch computed from 
a figure; (2) the cues arc interpreted in three space; (3) a surface is constructed that is most conservative with 
respect to meeting the constraints imposed by step (2). 


It follows that the perception of subjective surfaces amounts to no less than understanding the application 
to two dmii iHorril figures of the proce ses-of natural perception, at least to the level of deteunining u.ible 
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surfaces. In particular, we need to understand the way in which the elements of the primal sketch are inter¬ 
preted in three dimensions, and tire constraint that they impose on the possibly subjective surfaces of which 
they are a part. This is no small task! It includes, for example, the tradition of work aimed at understanding 
line drawings as three dimensional scenes [Clowes 1971, Barrow and Tencbaum-1981, Binford 1981, Draper 
1980, 1981, Huffman 1971, Mackworth 1973, Waltz 1975, Sugihara 1978,1981, Kanade 1981]. An interesting 
possibility for future research is the use of subjective contours as a means of furthering our understanding of 
tire interpretation of line drawings as three-dimensional surfaces. 

In this section we briefly discuss tire entities that-Mart [1976a] proposes are written into the primal sketch, 
and are interpreted in the manner outlined by Binford [1981]. The following subsections discuss dots (or small 
blobs), zero-crossing contours (for example, the bounding contours of figures), and line endings. These entities 
have been discussed in several papers in the subjective contour literature. 

In general, several surfaces can be irherpolated through a given set of constraints. As further constraint 
is added, some candidate surfaces are ruled out. It seems that the more tightly the surface is constrained, the 
greater its apparent clarity. Notice that the apparent brightness may not increase, even if die apparent clarity 
docs. We return to this point in the next section. Two further points are in order. First, the interpolation 
scheme that underlies our account of surface perception is capable of accepting a variety of cues. This should 
be contrasted with the schemes devised by Ullman [1976] and Horn [1981]. Second, the interpolation algo¬ 
rithm does not require the surface to meet the given constraints exactly. It incorporates a "penalty" scheme 
that allows some constraints to be met approximately. This allows local deviations from constraints to achieve 
flic best global surface, Grimson [198 la] gives details. 


Dots 


A dot or small blob in the primal sketch marks a point in three-dimensional space. Although its position 
in the image is determined, the depth from the viewer is not (Gibson [1979] refers to such a constrained point 
as an optic ray). In general, a dot may be in, on, or near a subjective contour, cor-esponding to a three-space 
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Figure 21. Dots may be in, on, or near, a subjective contour. 


point that is on the boundary of, contained in, or outside of, a subjective surface (see Figure 21). A dot that 
is interpreted as contained in a subjective surlacc appears as a mark that has different reflectance from its 
surrounds, like an ink blob (sec below also). Points that are interpreted as lying outside a subjective surface are 

not accorded significance. 

As we discussed above, if a dot is on or near a subjective contour, the contour is constrained to pass 
through it, and the subjective surface is seen more clearly (figure 22). Vertices and the ends of space curves 
also play a significant role in three-space interpretations of line drawings. Line endings are discussed later. 
By Grimson’s “no news is good news” dictum, our perceptual system expects such points to be made explicit. 
Figures 23 and 24 show the effect of adding points that mark vertices. Figure 24 is particularly interesting as 
the dots disambiguate the shape of the subjective surface, which is shown without the dots in figuie 24a and 
appears as a circle or a square. The circle versus square interpretation is also affected by the visual angle from 
which the figure is viewed. 


Blobs such as that shown in figure 25 appear as closed zero-crossing contours in the primal sketc h, and arc 
interpreted as surface patches or as object silhouettes [Mart' 19/7, Brady 1979]. 1 lie chance in intensity acioss 
such a blob lias been charm a ri.vd as a step (see for example Mari [1970a], i lerskmii:, and Binlord |1970|). In 






Figure 22. Dots can provide extra constraint on a subjective surface, leading to increased clarity 
of the subjective contour. . 



Figure 23. The addition of dots to mark the vertices of a subjective figure can greatly enhance 
its clarity. 


natural scenes, such intensity changes typically correspond to bounding contours or to reflectance boundaries, 
'[’his suggests that it is possible to generate subjective surfaces in which some blobs arc interpreted as surfaces 
occluded by the subjective surface and some are interpreted as reflectance patches. Examples of this sort have 
appeared in the literature, for example, Kanis/.a (1974, figure 34, page 109] (see figure 26). 







Figure 24. The addition of dots to mark the vertices of a subjective figure can rule out other 
possible figures by requiring the contour to pass through the dots. 



a b 



Figure 25. A blob is interpreted as a surface patch or as the bounding silhouette of an object, 
(a) A blob dial is not interpreted three-dimensionally. (b) A blob Unit is interpreted as a tilled 
three-dimensional surface, (c) A blob that is interpreted as dte silhouette of a three-dimensional 

object (alter Marr [1977]). 
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Figure 26. A “pinto” subjective triangle in which some blobs are interpreted as occluded by the 
subjective surface and others are interpreted as patches of surface color. 





Recall that dots constrain the position of a subjective surface. Those parts of a blob that arc interpreted as 
lying along the subjective contour impose additional constraint on its direction in the image. More precisely, 
they constrain the image of the tangent to the space curve that forms the subjective contour. In general, the 
foreshortening is not determined. In fact, our definition of a dot is a closed zero-crossing contour which is 
too small to have an accurate tangent direction computed for it. The additional constraint afforded by a blob 
produces dear subjective contours in cases where there is none when the blobs are replaced by dots (sec figure 
27). 


The intensity change associated with the blob is as important as the geometric shape of the blob. Figure 


28 shows that “empty” blobs arc not effective for generating subjective surfaces. The intensity change across 
the boundary lines of an “empty” blob is not step-like [Marr 1976a]. It might be objected that the concentric 


disks used in many demonstrations of subjective contours are not solid blobs, but are “empty” in some sense. 


Discussion of this point is deferred until the next section. 


There is, however, more to a blob than a step change in intensity. The geometrical shape of the blob 
and the descriptions computed by the visual system play a crucial role. The computational theory of vision. 
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Figure 27. (a) The four dots are grouped to form a parallelogram, but there is no strong three- 
dimensional percept, (b) Replacing the dots by blobs produces a tilted subjective surface, (c) If 
two of the blobs are replaced by dots, the subjective surface is not seen clearly. 




Figure 28. 


Empty blobs are 


not effective at generating subjective surfaces. 


Iced the entire subject of.pattern recognition, has not yet devised an adequate -account of shape descsiptioi 
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Figure 29. The blob in (b) is typically described as a blob like that shown in (a) with a piece 
cut out 


V . 


Consider figure 29. For this article, the crucial point is that figure 29b appears to be “incomplete”, to be a ver¬ 
sion of figure 29a from which a piece has been cut. Kanisza [1974] has argued that it is precisely descriptions of 
this sort that underlie the perception of subjective contours. An initial conjecture is that tangent discontinuities 
that produce concavities in otherwise smooth contours typically give rise to descriptions that entail missing 
parts. The qualifying phrase “otherwise smooth” recognizes that there are shapes such as figure 30 in which 
tangent discontinuities do not always signal incisions. Figure 31 shows that smooth blobs (of which the circles 
favored by Kanisza are merely a special case) are more effective for generating subjective surfaces than those 
in which there are several tangent discontinuities. Several authors have pointed out that extremely angular 
blobs are ineffective at generating subjective contours (Corcn [1972, figure 8c, page 365], Kanisza [1974, figures 
5, 6, page 96], Brigner and Gallagher [1974, figure 2, page 1049], Rock and Anson [1979, figure 3, page 667]). 
Angularity is neither a necessary nor sufficient condition, however. 


Lines 


:.4T \ 


Lines in an image give rise to connected contours of zero-crossings in the primal sketch. Binford has 

i v* ;• ■ • . . 

isolated the following list: (a) wires, space curves that arc not attached to any sir 'ace, (b) bounding surface 





Figure 30. Tangent discontinuities do not always lead to descriptions that refer to pieces being 
cut out. The shape shown here is often described as a rectangle with a piece stuck on its side. 




Figure 31. Smooth blobs with tangent discontinuities are more effective lh;m angular blobs for 
generating subjective surfaces. 

contour, curves that arc attached to one flanking surface and signal a depth boundary, (c) connect edges, 
curves that are connected u> the two surfaces that flank the curve, (d) curves that form the boundary of a 
surface marking, and (e) curves that mark the boundary of a shadow. In case (c), the connect edge may be 
cither convex or concave with respect to the viewer. 






Figure 32. A subjective contour with a protruding wire. 


To each threc-space interpretation of a line there is a subjective contour having that interpretation. In sec¬ 
tion 2 we presented examples of subjective surfaces that included bounding contours (the usual case), shadow 
boundaries (figures 16 and 17), and connect edges whose convexity or concavity was occasionally ambiguous' 
(figures 7, 8, 11, and 18). The only interpretation that was not demonstrated in section 2 was as a wire. That 

deficit is repaired in figure 32. 


As was the case with blobs in the previous subsection, the process by which an individual line in the 
image is interpreted is not completely understood. The best account to date is the tradition of work aimed 
at understanding line drawings of polyhedra, together with the more recent work of Barrow and Tenenbautn 
[1981], Binford [1981], and Stevens [1981c]. For example, we typically interpret a straight line as a straight 
space curve, and a curved line as a curved space curve. We prefer to interpret curves lying in a surface as lines 
of curvature (figures 3, 5, and 17) [see, for example, Stevens 1981c]. It is difficult to indicate non-zero torsion 
of a space curve by an isolated (curved) line, so strong is our preference for planai interpretations. A closed 
contour lying totally within a subjective contour is typically interpreted as a icflcctance houndaty (see figure 
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Figure 33. Closed contours enclosed by a subjective contour are typically interpreted as reflectance 
boundaries. (Reproduced from Kanisza [1974, figure 36, page 109]). 


Each interpretation of a line imposes constaint on die surface interpolation process. A straight line either 
lies in a subjective surface or is normal to the surface. Figure 34a shows a subjective tilted plane with wires 
that appear to be normal to it, whereas in figure 34b the wires appear to be lie in the plane. Kanade [1981] 
has considered die conditions under which three straight lines in an image can be the images of vectors in 
three orthogonal directions in space. It turns out that the more nearly two of the image lines are parallel, 
die more likely the two vectors are to interpreted as coplanar. This observation underlies Kennedy’s [1976a] 
“perpendicularity hypothesis”, which Parks [1980a] showed to be inadequate. Interestingly, the counter¬ 
examples presented by Parks involve lines that are collincar in the image and are perceived as a single space 
curve occluded by the subjective surface. The importance of line orientation is also shown by an example 
of Rock and Anson [1979, figure 7b, page 674], They report that the introduction of the stripes “completely 
eliminated the perception of an illusory contour”. The striped texture in Rock and Anson's figure is nearly 
collincar with one of the sides of the subjective triangle. Figure 35 shows that the subjective contour is still 
percei ved when it is not nearly collincar with the orientation of the texture. 

A line ending may be near a subjective contour, in which case the line is typically interpreted .is occluded 
by the subjective surface, tangent to the subjective surface, or .normal to it. A line ■ hose orientation is close to 





b 


Figure 34. (a) A tilted subjective planar surface with wires that are perceived as normal to it. 
(b) The same subjective surface with wires perceived as lying in it. 





Figure 35. A subjective contour despite strong background texture. 


that of a subjective contour constrains the subjective contour to lie along it. and hence constrains its tangent in 
much the same way as a blob (sec figure 36). 

In summary, although the interpretations are imperfectly understood, we attribute significance to the 
relationship between the orientation of a line and the underlying subjective surface. For example, figure 37 is 





Figure 36. The lines constrain the subjective contour to lie along it. 


due to Parks [1980, figure 2, page 362], The subjective surface in figure 37a is clear and the lines are interpreted 
as occluded by it. The subjective surface in figure 37b is less clear. There is no compelling interpretation of the 
relationship between the lines and the underlying surface. 


5. The role of apparent brightness 


It will not have escaped the attention of readers who are familiar with the literature of subjective contours 
that this article has been conspicuous in that it ignores what is for many researchers the most important, 
perhaps the determining, characteristic of subjective contours. We refer to the apparently enhanced brightness 
of the region enclosed by many subjective contours relative to the region outside the subjective contour. 
Broadly, there have been two explanations of this phenomenon, and they have largely determined the rcspec- 
tivc authors’ approaches to subjective contours as a whole. Some authors, notably Corcn [1972] have argued 
that the apparent brightness of die subjective surface is a consequence of tire three-dimensional interpretation 
of subjective contours as stratified planes parallel to the image plane. The subjective figure is perceptually 
nearer the viewer, and by some version of constancy scaling therefore appears brighter. Other authors, for 
'example Frishy and Clalworthy [1975], Brigner and Gallagher [1975], Kennedy and Fee [1 976], Jury and Ony 
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Figure 37. The dear subjective contour in (a) derives from the perceived relationship between the 
lines and the subjective surface, despite the fact that the lines in (b) are more nearly orthogonal 
to the subjective contour. (Reproduced from Parks [1980, figure 2, page 362].) 


[1979], Day and Jory [1978, 1980] prefer an explanation based on some version of lateral inhibition. Both 
.positions have weak points. Coren and ITteodor [1975], for example, point out that there are corners in one of 
their images that ought to give rise to an enhanced brightness effect, according to arguments based on lateral 
inhibition. Parks [1980b] shows how confounded the effects of brightness and depth are. 


In the previous sections we have argued that the processes of natural perception, misapplied to two- 
dimensional figures, are at tire heart of understanding subjective contours. We sketched the computational 
theory of visual perception pioneered by Marr, paying particular regard to two representations, die primal 
sketch and the 2|-D sketch. We noted that the 2^-D sketch makes explicit information about visible surfaces, 
including surface discontinuities, that is computed, whenever possible, from stereo, motion, perspective, tex¬ 
ture, occlusion, and so on. These processes, in turn, operate on the information made explicit in the primal 
sketch by the earlier operations of edge finding and grouping [Marr 1976a], To d uc, the complete structure 










Figure 38. Examples of the convolution of common triggering elements with a V 2 G filter at a 
variety of scales. The values demonstrate enhanced brightness, assimilation, and dissimilation, (a) 
The result of processing one of Kanisza’s figures with a fine filter, (b) The result of processing 
the same Kanisza figure with a coarse filter. 


of the primal sketch, and complete understanding of the processes of edge finding and grouping arc not avail¬ 
able, but a solid start has been made, particularly in the zero-crossing theory of Marr and Hildreth [1980, also 
Hildreth, 1980]. The theory proposes that lateral inhibition consists of convol ving an image with operators that 
are the Laplacian of a Gaussian distribution, operators that closely approximate die difference of Gaussians 
suggested by Wilson and Bergen [1979] (see also Richter and Ullman [1980].) 


By our account, not ail edges are created equal in the real world, and it is the job of early processing to 
discover descriptions of contrast, orientation, finely separated lines, blobs, and local textmes. 1 hese processes 
inevitably discover differences in the two-dimensional triggering conditions discussed above and by other 
authors. Figure 38 shows the result of convolving a number of common triggering elements at a variety of 
scales. I he results are preliminary, and will be the subject of a future report, but they clearly indk.ite some 
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of the enhanced brightness effects suggested in the literature, such as Frisby and Clatworthy’s claim that line 
endings yield larger enhanced brightness effects than the sides of lines, Jory and Day’s distinction between 
assimilation and dissimilation effects, and Smith and Over’s finding that subjective contours produce tilt and 
motion after-effects. 


We argued above that the figural completion problem studied by Kanisza formed an important part of 
the perception of subjective surfaces, but that it pertained to a stage of processing that is earlier than the 
computation of visible surfaces. Similarly, edge detection, with its concomitant brightness effects plays an 
important part in computing the primal sketch. This is, however, also an earlier stage of processing than the 
determination of visible surfaces. By our account, both enhanced brightness effects that stem from lateral 
inhibition and the brightness that results from constancy scaling have a part to play in the perception of 
subjective surfaces. 


7. Conclusion 


It was proposed that subjective contours are an artifact of the perception of natural three-dimensional 
surfaces. A recent theory of surface interpolation implies that “subjective surfaces” are constructed in the 
visual system by interpolation between three-dimensional values arising from interpretation of a variety of 
surface cues. We showed that subjective surfaces can take any form, including singly and doubly curved 
surfaces, as well as the commonly discussed fronto-parallel planes. In addition, it is necessary in the context 
of computational vision to make explicit the discontinuities, both in depth and in surface orientation, in the 
surfaces constructed by interpolation. It was proposed that subjective contours foim the boundaries of die 
subjective surfaces due to these discontinuities. Several novel subjective surfaces and subjective contours were 
demonstrated to support this position. The role played by figure completion and enhanced brightness contrast 
in the determination of subject ive surfaces was discussed. 
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